Repository navigation
Conversation
libmagic's description for GNU message catalog (.mo) files can include the gettext PO header's Project-Id-Version field verbatim, which is free-form content taken from inside the scanned file (translator comments, escaped non-ASCII text, etc.) rather than structural type information. This leaked raw file content into filetype_file and anything derived from it. Strip that trailing Project-Id-Version content in _detect() right after decoding libmagic's output, keeping only the structural 'GNU message catalog (...), revision X, N messages' description. Added tests/test_magic2.py cases covering both leaked examples from the issue (plain text and escaped-byte Project-Id-Version values) and confirming unrelated file types are left untouched. Fixes aboutcode-org#43 Signed-off-by: Jitendra Bansal <jitendrabansal0300@gmail.com>
Author
|
The |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #43.
For
.mofiles (GNU message catalogs), libmagic's detected type stringcan include the gettext PO header's
Project-Id-Versionfield verbatim,e.g.:
That trailing part after
Project-Id-Version:is free-form content takendirectly from inside the scanned file (translator comments, escaped
non-ASCII bytes, etc.), not structural type information, so it was
leaking into
filetype_fileand anything derived from it (as reportedagainst the two Django
.mofiles linked in the issue).Change
In
magic2._detect(), right after decoding and collapsing whitespace onlibmagic's raw output, strip any trailing
Project-Id-Version: ...thatfollows the structural
GNU message catalog (...), revision X, N messagesdescription, via a regex matched only against that specific pattern so no
other file type strings are affected.
Tests
Added
tests/test_magic2.pycases that exercise the stripping regexdirectly against both leaked strings quoted in the issue (the plain-text
one and the escaped-byte one) and confirm unrelated type strings
(
ASCII text, an ELF description) pass through unchanged.Verification note
This sandbox doesn't have a working
typecode-libmagicplugin install(module import fails before even reaching this code, on an unrelated
plugincodeentry-point issue in this environment), so I couldn't runfile_type()end-to-end against real.mofiles here. I verified theregex logic in isolation against the exact strings quoted in the issue
(including the escaped-byte example) and against unrelated type strings,
and the new tests encode that same verification so they'll run in CI.